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ABSTRACT 

A continuous pupil progress monitoring system was 
implemented in two elementary schools; 552 students and 38 
educational personnel were involved. , The monitoring system employed 
was initially designed to evaluate students receiving special 
services and their progress toward individual education plan reading 
qoals. The model emphasized direct assessment in the students grade 
level basal reader (Ginn Reading Series) and repeated measurement of 
the number of words read correctly in students' 1-minute readings for- 
16 weeks. The number of words read correctly by the. student was 
tabulated and the information entered into a microcomputer program. A 
computer printout displayed a graph of each student's performance and 
a summary of descriptive statistics (baseline level, current level, 
average weekly gain or loss, and amount of variability in reading 
scores). Actual time- for administration and scoring was approximately 
3 minutes per student. Teachers generally felt that the information 
collected was instructional^ relevant. The extent to which student 
performance data were valid indices of student achievement, were 
sensitive to pupil growth, and could be used to judge the efficacy of 
program placement and student progress in the program also were 
examined. Results suggest that the system can be implemented 
successfully on a wide-scale basis in schools. (Author/PN) 
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Abstract 

A continuous pupil progress monitoring system was implemented in 
two elementary schools; 552 students and 38 educational personnel were 
involved. The educational personnel provicjfd input on the feasibility 
and cost effectiveness- of the system and their reactions to the 
system. The extent to which student performance data were valid 
indices of student achievement, were sensitive to pupil growth, and 
could be used to judge the efficacy of program placement and student 
progress in the program also were examined. All results suggested 
that the system can be implemented successfully on a wido-stale basis 
in schools. The need for commitment on the part of the school 
administration is discussed. 



Implementation of fHrect and Repeated Measurement in 
the School Setting 

* 4* 

In an era of declining enrollments, cuts in state and federal 
aids, and a resurgence in private education, the public schools are in 
a position where they must justify their appropriations. Minneapolis 
Public Schools has responded to this challenge by developing and 
implementing a Five Year Plan that outlines how the district will meet 
the present and future needs of their students in our changing 
society. Essential to this plan is the ability to document the 
district's effectiveness. 

A significant goal identified in the long-range plan involves the 
need to monitor student progress. 

Accountability fbr\^ student achievement of basic skills 
requires development of a system-wide data-based management 
system for monitoring student progress. The district will 
design and implement a stadent achievement data system. 
(Minneapolis Public Schools, p. 14) 

in an effort to achieve this stated objective, the Minneapolis 
Public Schools and the Institute for Research on Learning Disabilities 
(IRLD) cooperated on a pilot project in which pupil progress in 
reading was monitored frequently. 

The monitoring system employed was designed initially to evaluate 
students receiving special services and their reading progress toward 
Individual Edj^rtional Plan (IEP) goals (Deno & Mirkin^ 1977; Jenkins, 
Deno, & Mirkin, 1979). The proposed model emphasized direct 
assessment in the students 1 curriculum and repeated measurement. IRLD 
investigators conducted a search for measures of reading that could be 

used in a classroom progress measurement system. To be considered for 

* * ** 

inclusion in the system, the reading measure had to have the following 
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characteristics: 

(1) Valid with respect to widely used measures of achievement 
in reading ♦ 

(2') Immediately sensitive to the effects of relatively small 
adjustments in instructional interventions 

(3) Reliable 

(4) Easy to administer by teachers, parents, and students 

(5) Have many parallel forms that are frequently administrable 
to the same student 

(6) Time efficient 

(7) Inexpensive to produce 

(8) Unobtrusive with respect to routine instruction 

(9) Simple to teach to teachers, parents, and children 

On the basis of several research study results, it was concluded that 

'the best measure of reading for the direct, repeated measurement 

system was the number of words read correctly by a pupil in one minute 

from his/her basal reader (Deno, Marston, Mirkin, Lowry, Sindelar, % 

Jenkins, 1982; Deno, Mirkin, it Chiang, 1982; Fuchs 4 Deno, 1981). 

Using the research base established at the IRLD, an attempt was 

made to develop a similar pupil progress monitoring system in two 

elementary schools. During the implementation of the pilot project, 

f ive'specif ic questions were addressed: 

(1) What is the feasibility and cost effectiveness of 
designing and maintaining a large scale student 
monitoring system in an elementary schbol? 

.(2) What are the attitudes df teachers and administrators 
toward such a system after its implementation? 

(3) Do student performance data validly index student 
achievement in this academic area? 

(4) Are student performance data sensitive to pupil growth? 



( 



(5) Can the student performance data be used to judge the 
efficacy of program placement ^nd student progress in 
those programs? « 

These five questions were the focus of the present research. 

Method 

Subjects ' 

The continuous pupil progress monitoring system was instituted 
for half a year in two elementary schools and involved 552 students 
and 38 educational personnel. School A was a K-3 elementary school; 
all 325 pupils in grades 1 through 3 participated in* the project. In 
School B, a K-6 elementary school, 227 students participated; these 
were students of teachers who volunteered to participate in the pilot 
project. 
Procedure 

Pupil evaluation in < reading began in late January 19)32 and 
continued for 16 weeks. Each week the students would \read 
individually to a teacher, tutor, or aid^ from their grade level basal 
reader for one minute.' AIT students read from the Ginn Reading Series 
^ (Clymer & Fenn, 1979). First graders read passages from Level 5, 
second graders read passages from Level 7, and third grade pupils read 
from Level 9. 

To reduce error in measurement, all passages were screened to 
ensure equivalence of passages within levels. The Fry Readability 
Index (1975) was used to assess reading levels of the passages. Any 
passage rated over one year above or below grade level was not used in 
the study. Average readability for Ginn 5 passages was 1.1; for Ginn 
7 the average readability was 2.6, and for Ginn 9 it was 3.3. 



At the ,end of. the timing, the number of words read correctly by 
the student was tabulated and the information entered into a micro- 
computer program. Figure 1 illustrates a^sample reading sheet scored 
for correct words. Teachers then were supplied with a computer 
printout displaying a graph .0^ each student's performance. Also 
included* was a summary of the following descriptive statistics: 
baseline level, current level, average Weekly gain (or loss), and 
amount of" variabil ity in reading scores. At the ervd of the four-m6nth 
period the mean and slope of words read correctly was calculated for 
each student,. 

Ins#t Figure 1 about here 



Results 

/ 

Feasibility and Cost Effectiveness 

A total of 38 educational personnel were involved in the weekly 

• * * 

measurement of the 552 pupils. Those working with the students 

included teachers, tutors, aides, a school psychologist, and a 

principal. A frequency analysis of the participants by job 

description is shown in, Table 1, 

Insert Table 1 about hare 



A1J 38 participants were asked* to complete a questionnaire 
surveying their attitudes* toward' the p'roject;^ included on the 
questionnaire was an item" asking them to estimate the amount of time 
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they spent administering and scoring the reading procedures. A copy 
of the* Teacher Evaluation Questionnaire is found in Appendix A. 
Twenty-five of the participants responded to the survey. The average 
number of students tested by the respondents was 15.5. The average 
amount of time needed ^to test and score the reading samples was 47.75 
.minutes, or approximately 3.0 minutes per student. * ^ 

Not induced in this efficiency analysis is the amount of time 
required for developing measurement materials, training teachers, and 
mon i tor i ng the data* co 1 1 ecti oh process . Both pi lot school s were 
Serviced by a school psychologist who fulfilled these responsibilities 
with about 40% of full-time effort. In addition, it was estimated 
that 10 hours/weetr of ^clerk time was required for entering data into 
the micro-computer. - ' 
Teacher and Administrator Attitudes 

The Teacher Questionnaire also solicited the opinions of the 
participants about the data moil i tori ngvs^stem. An analysis of how the 
participants used the data is presented in Table 2. The majority of 
teachers (72%) found the data\j$#ul for tracking student progress in 
reading. Some teachers felt the system was helpful in communicating 
.with parents (32%) and other teachers (32%). For example, several 
teachers used the computer printout during parent-teacher conferences; 
they reported that many parents responded favorably to their child's 
graphed data. Nine of the twenty-five respondents (36%) stated that 
the information was useful for instructional planning. The most 
frequently cited example was teacher reevaluation of student placement 
in reading groups. Others commented that they changed students' 
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reading placements based on the data*. Finally, 28% of the respondents 
remarked that 'their expectations about student achievement potential 
had changed. In one case, a teacher referred a student to special 
education; she had not previously been aware that the student was 
having problems In other cases, .students did betted than teachers 
had expected. * : 



Insert Table 2 about here 



Participants also were asked to -answer questio.ns about the 
efficacy of the daj:a system in general (see Table 3). When asked 

s 

whether Minneapolis Public Schools should continue this type of system 

> » •* 

.to track student progress, 68% responded positively, while 20% said 
no. With respect to participant beliefs about whether trends in the 
data reflected actual' student progress, over one-half (56%) of the 
participants answered" yes, while 16% responded no. - Finally, 80% of 
the participants evaluated the organization of -* the measurement 
"materials as sufficient. 



Insert Table 3 about here 



Validity of Measurement System 

' Investigating the validity of the student reading data is 

i . , 

essentially a replication study. Deno, Mirkin, & Chiang (1982) 
demonstrated earlier that the number of words read correctly during a 
ona-minute timing from a basal reader correlates highly with 



•performance on the Stanford -Diagnostic Reading Test (Karlsen, Madden, 
& Gardner, 1977), the Woodcock Reading Mastery Test (Woodcock, 1973), 
•and. the Peabody Individual Achievement Test (Dunn & Markwardt, 1970). 

i 

For this study, several test§ were administered to a group of 26 
thMnl grade students. The criterion measures were the reading subtest 
scores from the (a) Stanford 'Achievement Te st (Madden, Gardner', 
Rudman, Karlsen*, & Merwih, 1973): Vocabulary, Reading - Words, 
Comprehension, Total Reading and Word Study, (b) the reading portion 
of the SRA A chievement Series (Naslund, Thorpe, & Lef ever, 1978): 

* — » ■ ■ ■ ^ 

Vocabulary and Comprehension, and (c) the Ginn 720 Reading Series 
(Clymer & Fenn, 1979). Subtest scores were correlated with the number 
of words read correctly by each" student from the basal reader. The 
obtained coefficients are displayed in Table 4. With the exception of 
Vocabulary, the coefficients were of high magnitude and ranged between 
".80 and .90. 



Insert Table 4 about here 



A second approach to examining the validity of Words Read 
Correctly is to compare it with other standardized tests of reading as 
"to their correlations with an external criterion. In this analysis, 
teachers were asked to judge student achievement* level in reading on a 
scale from 1 to 5 (Teacher Judgment). Performance on the direct 
measure of reading and on the standardized test's then were correlated 
with the Teacher Judgment variable. These coefficients are ^r^ented 
in Table 5. Words Read Cowectly correlated .77 with Teacher Judgment 



and was* the second largest coefficient. Coefficients for the 

» 

remaining standardized tests ranged from .47 to .81. The -.77 
coefficient was then .compared to the remaining coefficients by 
examining the significance of the difference between two correlation 
coefficients for correlated samples (Ferguson/ 1971) . The coefficient 
for Words Read Correctly was significantly 'greater than SRA 
Vocabulary, SAT Vocabulary, and Ginn Reading Level. 

i 

* Insert Table -5 about here 



Sensitivity to Pupil Growth 

Verification o* an assessment procedure, as sensitive to pupil 
progress is dependent upon an external measure or criterion. Teacher 

ratings, of student performance again were used as the criterion 

«, 

variable in the analysis of sensitivity. The same 26 third-grade 

* "* 
students used in the validity analysis were rated on a scale from 1 to 

5 T)> the teacher in terms of how -well- the student had .progres.s.ed_in 

reading during the four-month period. Progress measures derived from 

the standardized tests and the direct measure of reading then were 

correlated with the Teacher Judgment score. It was reasoned that 

those measures correlating highest with Teacher Judgment were the most 

sensitive to growth. For Words Read Correctly, the progress measure 

"was the difference between the mean of the f.irst three weeks 

subtracted from the mean of weeks 14, 15, and 16. For the^ SRA 

Vocabulary and Comprehension Tests, which were administered at Weeks 1 

and 16, the difference scores were used to monitor student growth. 



Although Ginn basal reading levels were available at the beginning and 
end of the four-month peridd, they were not used in the analysis 
because of teacher inconsistencies in using mastery criteria for 
moving students through the readers. For example, in third grade, 
where 749 unit mastery tests were given in the four-month period, 
students failed to meet criterion on 262 tests (35X); All pupils had 
been advanced to higher reading levels despite failing the criteria on 
the unit mastery tests. 

Correlation coefficients between Teacher Judgment and the 
dependent measures are presented in Table 6. The Words Read Correctly 
measure showed is the highest, correlation, .43. This coefficient was 
significantly greater than the .01 coefficient for SRA Vocabulary, but 
was not significantly different when compared to SRA Comprehension 
(Ferguson, 1971). While the low magnitude of the coefficients may be 
surprising, this may be partially due to the lowered reliability of 
difference scores (Thorndike & Hagen, 1978). 



Insert Table 6 about here 



A second approach to analyzing sensitivity to growth is to 
determine the extent to which students progressed on the' various 
measures. Correlated t_ test analyses were used to analyze the amount 
of growth on Words Read Correctly, SRA Vocabulary, and SRA 
Comprehension between the initial and- later stages of the study. 
Larger t values are indicative of greater change or growth. Means, 
standard deviations, and t values for this analysis are presented in 



Table 7. Change was most evident for Words Read Correctly (_t = 6.65, 
£ < .001) and least for SRA Comprehension (_t = 1.65, £ ■ .113). 



Insert Table 7 about here 



Judging Program Placement and Efficacy 

This analysis was confined to School A, where students were 
enrolled in grades 1-3. Three types of program placement were 
analyzed: 'regular education (N - 130), Title I service (It = 104) ? and 
special .education (N = 24). The analysis conducted examines the mean 
number of Words Read Correctly of the students in each program. If 
this measure is useful for determining eligibility, it should reliably 
differentiate student? placed in the various services. The mean 
number of Words Read Correctly^ for each level of service at the 
different grade levels is reported in Table 8. The Total Sample 
means, also found in Table 8, were subjected to a one-way analysis of 
variance with program placement functioning as the independent 
variable in the analysis. 



Insert Table 8 about here 



Results of the ANOVA were highly significant, £(2,269) = 111.8, £ 
< .001. A Student-Newman-Keuls follow-up test was used to compare the 
placement group means. The mean for regular education placement was 
significantly greater (£ < .05) than Title I and special education 
means. Title I ancJ special education, however, did not differ 
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significantly. Comparison of these two groups at each grade level, 
however, showed significant differences at grade 2 and grade 3. 

Finally, an attempt was made to judge the efficacy of each level 
of service by examining student learning rate. The slope of each 
pupil's performance was computed for weeks 1 to 16. The mean slope at 
each grade level for the different levels of service is presented in 
Table 9. Substantial differences are apparent. For example, the 
average first grader in regular education made gains at a rate of, 
about 4.4 words per week, whereas the average first grade special 
education student made about a 1.6 word gain. 



Insert Table 9 about here 



The analysis of the data, however, focused on mean slope within 
each level of service across grades. Total sample mean sjope for 
regular education wa* 2.8; it was 2.8 for Title I and 1.7 words per 
week increase for special education. ANOVA results indicated a 
significant difference among means, £(2,255) 35 3.70, £ = .026. The 
follow-up test indicated that the mean learning rate of both regular 
education and Title I students was significantly greater than special 
education students. Regular education and Title I did not differ 
significantly from each other. 

One question arising from these results is: How do learning 
rates of special education students differ from low-functioning 
students placed in regular education and Title I? Those students in 
regular education and Title I services who read at or below the 
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special education mean for, their grade level were identified as 
M low-achi6ving. H The learning rates of low achievers were then 
compared to the special education students. The mean slope for "low 
achievers" was 1.66 with a standard deviation of 1.20. Special 
^ education students had a mean of to. 72 and a standard deviation of .97. 
The two means are not rel iably different (_t = -.22, £ = .824). 

Discuss io ; n 

Implementation of wide scale change in any organization is 

* v 

probably best served if the process is examined systematically. This 
approach has been adopted by MinneapoHs— Publ ic Schools in their 
. attempt to design a system for monitoring student progress. The 
present study described the .implementation a progress monitoring 
system in two elementary schools. 

The quality of an assessment system is frequently measured in 
terms of technical adequacy (Salvia & Ysseldyke, 1981). Lowever, 
demonstrating that specific assessment procedures are valid and 
reliable does not totally substantiate the worth of an assessment 
methodology. Although the importance of technical characteristics is 
acknowledged and presented in this report, we have endeavored to look 
beyond these qualities, examining the efficiency of the proposed model 
and teacher attitudes toward it. As a result, general statements 
regarding the utility of the system may be made. 

A first issue addressed was that of how much personnel and time 
commitment was necessary. Actual time required for administration and 
scoring was quite low, approximately three minutes per student. Given 
the recent research on academic responding time, the trade-off is not 
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unreasonable. * Graden, Thurlow, and Ysseldyke (1982), for example, 
found the average time allotted to students for reading aloud was 0.7 
minutes a day. Reading from a basal reader once per week for one 
minute would represent a 48< increase. 

Teacher attitudes toward the system also were examined. On all 
questions, a majority of the respondents favorably evaluated the 
progress measurement system. In some respects th is finding is 
surprising since over half of the participating \ personnel were 
required to participate. Although several teachers initially objected 
to their involvement, there seerned to be a change in attitude by the 
end of the project. In fact, there was a general feeling among 
teachers that the information collected was instructional!^ relevant. 
This appears to be a positive characteristic viewed in light of 
teachers 1 low regard for the instructional usefulness of standardized 
tests (Thurlow & Ysseldyke, 1982). 

Principals at both of the schools involved in the project 
supported the system, with one asking teachers to bring student 
reading graphs to Student Support Team (SST) meetings. The same 
principal also had graphs sent home with student report cards at the 
end of the year. 

« 

The technical adequacy of the measurement system in terms of both 
validity and sensitivity to growth also was documented. Words Read 
Correctly proved to be both highly valid and the best measure of 
student progress. The evidence serves to further reinforce the notion 
that a progress monitoring system based on direct and repeated 
measurement is feasible. 
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> 

Finally, the results of the present study indicated that direct 
and repeated measurement may , be used for more than progress 
measurement. Salvia and Ysseldyke (1981) delineate five purposes of 
assessment: screening, identification, program planning, program 
monitoring, and program evaluation. The data presented here 
concerning the placement of students suggests that direct and repeated 
measurement could be employed for eligibility decisions and efficacy 
research. Marston, Tindal, and Deno (1982) have suggested that the 

use of a common data base across all five decision-making areas would 

i m 

be advantageous because it would increase efficiency, improve 
communication, and be related more closely to the skill areas for 
which schools provide instruction. 

In summary, the data suggest that direct and repeated measurement 
systems can be implemented on a wide-scale basis in our schools. The 
optimism must be tempered by the need for thorough organization and 
strong commitment from administrative levels. Given these conditions, 

if 

direct and repeated measurement should be viewed as a viable 
alternative for monitoring student progress in the schools. 
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* * Table 1 

Frequency Analysis of the Number of Participating Educational 
Personnel by Job Description 





Total Number 


Job Description 


Participating 


Teacher - Regular Education 


24 


\ 




Teacher - Special Education 


4 


Tutor 

* 


2 


Aide 


6 


School Psychologist 


1 


Principal 


1 


Total 


38 



2j 
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Table 2 

How Teachers Used the Information Provided in the 
Continuous Evaluation. Reading Project 



a ■ ' 1 

Percentage of Teachers 
Use of Data Using Data 



. i . 

Monitoring student progress in reading 


72% 


Communicating student progress to parents 


32% 


Communicating student progress to other teachers 


' 32% 


Planning for the instructional program 


36% 


Changed expectations regarding students 1 


28% 


achievement potential 





* 



Table 3 

Attitudes Toward the Continuous Measurement System 



Not No 
Yes No- Sure Response 



Do you think a system like this should 68°/, 20% 0% a \l% 
be used by Minneapolis Public School o 
teachers to tr*ck growth? 

When there is a trend in the data 56% 16% 16%' 12% 

. (increase or decrease) does it 
reflect the student's growth? 

Was the organization of the materials 80% 0% , 0% 20% 
sufficient? 
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k Table 4 

Criterion Validity Coefficients for Words Read Correctly 



Words Read 
Correctly 







Stanford 




Ginn 


SRA 


SRA 


Achievement 


Test 


Reading 


Vocabulary 


Comprehension 


1 2 3 


4 5 


Level 


.80 


.80 


.59 .84 .88 


.90 .84 


.83 



1 Vocabulary 

2 Readinq Words 

3 Comprehension 

4 Total Reading 

5 Word Study 



1 



i 



x Table 5'* 



Comparison of Words Read Correctly with. Other Measures of Reading 
^ p For Correlation with Teacher Judgment 

TeaCher Judgment 



Words Read Correctly - '77 

SRA - Vocabulary' , .63 

SRA - Comprehension .81 
Stanford Achievement Test 

f Vocabulary .47 

Reading Words .68 

• Comprehension , .75 

Total Reading /.74 

Word Study .68 

Ginn Reading Level ,.56 
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Table 6 

Correlation of Teacher Judgment of Progress with 
Measures^ Reading Progress 



Teacher Judgment 



Words Read Correctly -43 
SRA - Vocabulary -01 
SRA - Comprehension -36 
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Table 7 

Comparison of Weeks 1 and 16 with Correlated T Test Analysis 



Measure Mean .' S.O. Prob 

Words Read Correctly * 

Week 1 72.5 29.5 6.65 .000 

Week 16 89.7 32.6 
SRA Vocabulary 

Week 1 14.5 6.4 3.23 .004 

Week 16 16.9 7.3 
SRA Comprehension 

Week 1 15.2 6.1 1 .65 .113 



Week 16 ' 16.5 5.9 



*Week 1 is the mean of weeks 1, 2, and 3. 
Week 16 is the mean of weeks 14, 15, and 16. 
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Table 8 

Mean Words Read Correctly for Each Level of Service by 

Grade Level 



Mean Mean Mean 

Grade Regular Education Title I Students Special Education 



1 62.6 40.2 28.9 

2 . 93.3 36.9 22.9 

3 114.6 64.8 49.3 

i 

To'ta) Sample 98.7% . 45.5 " 36.5 
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Table 9 

Mean Slope of Words Read Correctly for Each Level of 
Service by" Grade Level 



Grade 


Mean 

Regular Education 


Mean 

Title I Students 


Mean 

Special Education- 


1 


4.4 


2.7 


1.6 


2 


2.7 


3.9 


1.4 


3 


2.6 . 


1.9 


1.9 


Total 


Sample 2.8 


2.8 


r.7 



" Eat a^!y! '* said (^y Mouse. 
" You will like this food. " 

Comlry Mouse said, " I do like it. 
] may nol go back to the co^itiy. " 

City Mouse said, " Don't go back! 
You can live, here with me. " 

When they were e^ng, 
City Mouse saw som^hi no/big. 

He* said r " Run! Run, Country Mouse. 

And don't stop 

Away went City Mouse. 
And away went Country Mouse. 
They ran out of the house. 
City Mouse called. " Come back, 
Country Mouse! 
There""is no danger now. 
The- cat went back into \hc house. " 

But Counliy Mouse did not stop 

He called. ** No, ) don't like to live 
where there is dangei 

Tm going home. " 

Country Mouse ran up a hill 
and into the country, 

* 

When he got home, he said, 
" At last I can stop! ^- 

Figure 1. Sample Reading Sheet Scored for Correct Words 
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Name 


31 




37 


Correct 


43 




MG 




52 




55 








CM 




70 




75 




7 7 


• 


87 




89 


• 


nr. 




]03 




107 


• 


110 




116 




170 




126 




131 
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.Appendix A 

Teacher Evaluation Questionnaire 
Continuous Evaluation Reading Project ; 

how* man^ students did you work with in the project? 



Approximately how much time did you spend each week administering 
and scoring the measures? <— 



How did you use the information provided by the continuous Evaluation 
Reading Project? , .{v 

(check) » t 

monitoring student progress in reading 

communicating student progress to parents 

communicating student progress to other teachers 

planning for the instructional program > 



changeo expectations regarding students' achievement 

potential 
other (describe 



Explain any responses: 
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A-2 



Do yuu think a system- like this should be used by Minneapolis Public 
School teachers to track growth? Why or why not? 



Dther tsub^ject areas? (check) 

math 

; social studies 

science 

(other) ' 



What did you like about this monitoring system? 



What did you not -ike about this monitoring system? 



What would you recommend to improve the system if it were to be 
expanded in its use? 
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9 

ERIC 



When there is a trend in the data (increase or decrease) does it reflect 
the student 1 s growth? 



Was the organization of the materials sufficient? How would you change 
the format? 



i 

Other commnets: 

r 
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